Papers with AI evaluation
How Many Ratings per Item are Necessary for Reliable Significance Testing? (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing methods for estimating model reliability are based on a few output responses per item. |
| Approach: | They propose a method to determine whether an existing dataset has enough responses per item to assure reliable null hypothesis statistical testing. |
| Outcome: | The proposed method can help researchers make better decisions about how to collect data for AI evaluation. |
Do Before You Judge: Self-Reference as a Pathway to Better LLM Evaluation (2025.findings-emnlp)
Copied to clipboard
| Challenge: | LLM-as-Judge frameworks are increasingly popular for AI evaluation, yet research findings on the relationship between models’ generation and judgment abilities remain inconsistent. |
| Approach: | They propose a self-reference-guided evaluation strategy that leverages a model’s own answers as references to strengthen the correlation between generation and judgment abilities. |
| Outcome: | The proposed approach strengthens the correlation between model generation and judgment abilities and provides a reliable proxy for model selection in evaluation tasks. |
A Comprehensive Framework to Operationalize Social Stereotypes for Responsible AI Evaluations (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent years have seen unprecedented gains in generative AI models' capabilities across modalitieslanguage, image, audio, and video domains across the globe. |
| Approach: | They propose a framework to operationalize stereotypes in generative AI evaluations using social psychological research and NLP data. |
| Outcome: | The proposed framework identifies key components of stereotypes that are crucial in AI evaluation, including the target group, associated attribute, relationship characteristics, perceiving group, and context. |